Stance Detection Benchmark: How Robust is Your Stance Detection?

Abstract Stance detection (StD) aims to detect an author’s stance towards a certain topic and has become key component in applications like fake news detection, claim validation, or argument search. However, while is easily detected by humans, machine learning (ML) models are clearly falling short of this task. Given the major differences dataset sizes framing StD (e.g. number classes inputs), ML trained on single usually generalize poorly other domains. Hence, we introduce benchmark that allows compare against wide variety heterogeneous datasets evaluate them for generalizability robustness. Moreover, framework designed easy integration new probing methods Amongst several baseline models, define model learns from all ten various domains multi-dataset (MDL) setting present state-of-the-art results five datasets. Yet, still perform well below human capabilities even simple perturbations original test samples (adversarial attacks) severely hurt performance MDL models. Deeper investigation suggests overfitting biases as main reason decreased Our analysis emphasizes need focus robustness de-biasing strategies multi-task approaches. To foster research important topic, release splits, code, fine-tuned weights.